[recipe, doc] feat: add Qwen-Image DPO v1 sync recipe - #536
Merged
knlnguyen1802 merged 2 commits intoSep 7, 2026
Conversation
Add a mechanical CUDA v1 sync recipe so online DPO can opt into TransferQueue without changing adapters or losses. AI assistance (Cursor) was used for this change. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
… path The v1 PG loop recomputed old_log_probs, but DPO infer returns noise_pred (log_probs=None). Route trainer_type=direct_preference through pairing + ref noise pred, matching v0. AI assistance (Cursor) was used for this change. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
knlnguyen1802
requested review from
AndyZhou952,
SamitHuang,
wtomin and
zhtmike
as code owners
September 4, 2026 09:23
Collaborator
Author
|
Reward curve |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
examples/dpo_trainer/qwen_image/run_qwen_image_online_dpo_lora_v1.sh, the CUDA v1 sync counterpart ofrun_qwen_image_online_dpo_lora.sh.main_diffusion→main_diffusion_v1,trainer.use_v1=true,trainer.v1.trainer_mode=sync, andtrainer.experiment_name=qwen_image_online_dpo_lora_v1. Model, LoRA, reward, pipeline, batch, and GPU layout are unchanged from v0.examples/dpo_trainer/README.md. The NPU script stays on v0 (same split as Wan2.2 [3/N][recipe] feat: add Wan2.2 DanceGRPO v1 sync recipe #466).verl_omni/pipelines/qwen_image_dpo/.This claims the Qwen-Image + online DPO · sync first row on #389. GPU layout is a mechanical clone of the v0 4-GPU colocated actor/rollout/reward recipe (
REWARD_TP=4onNUM_GPUS_ACTOR_ROLLOUT_REWARD=4). Aseparate_asyncrecipe is not included: that would need a dedicated rollout pool and is not a mechanical clone.Cloned from the landed diffusion v1 sync pattern in
examples/flowgrpo_trainer/sd35/run_sd35_medium_ocr_lora_v1.shandexamples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh(#466).Note:
main_diffusion_v1selectsPolicyGradientDiffusionTrainerV1Sync. The v0 online DPO path usesDirectPreferenceRayTrainerfor top/bottom pairing and trainer-sideref_noise_pred. This PR does not add a second trainer; it is the recipe-only switch requested by #389. Please treat a first GPU smoke as the check that pairing / ref-noise still flow through the shared DPO loss on v1.Checklist Before Starting
[{modules}] {type}: {description}Why this is not duplicating an existing PR
gh issue view 389 --repo verl-project/verl-omni --comments— tracking issue is open; Qwen-Image + online DPO is still in Need help / unclaimed. Comments claim BAGEL FlowGRPO ([recipe, tests, doc] feat: add BAGEL FlowGRPO V1 sync recipe #511) and MixGRPO (Add V1 sync recipe for Qwen-Image + MixGRPO #512), not online DPO.gh pr list --repo verl-project/verl-omni --state open --search "389 in:body"— no open PR for Qwen-Image online DPO.gh pr list --repo verl-project/verl-omni --state open --search "v1"/"DPO"— no matching online-DPO v1 recipe PR.gh pr list --repo knlnguyen1802/verl-omni --state all --search "dpo"— no matching recipe PR on this fork.examples/dpo_trainer/sd35/) stays on v0 by design and is not in this PR.Test plan
python tests/special_sanity/check_pr_title.pywithPR_TITLE='[recipe] feat: add Qwen-Image online DPO v1 sync recipe'— valid (modules: recipe;type: feat).bash -n examples/dpo_trainer/qwen_image/run_qwen_image_online_dpo_lora_v1.sh— OK.run_qwen_image_online_dpo_lora.sh: only header, entrypoint, v1 flags, andexperiment_namesuffix differ.bash examples/dpo_trainer/qwen_image/run_qwen_image_online_dpo_lora_v1.shon a 4-GPU node before merge.AI assistance
AI assistance (Cursor) was used to draft the recipe clone, README mention, and this PR. A human submitter has reviewed every changed line and is responsible for the change.